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Amendments to the Specification 

1 . Please delete the paragraph appearing on page 1 , lines 8-12 which read: 
Cro ss Reference to Related Applicati ons* 

This application claims priority to United Stat e s Pat e nt Application Serial No, 
60/197,971, which application is h e rein incorporat e d by r e f e r e nc e in its entirety. This 
application is a continuation of PCT/USO 1/0 4 963, which application is h e r e in incorporat e d 
by r e f e r e nce in its e ntir e ty. 

2. Please add the following paragraph immediately before the paragraph 
commencing on page 1 , line 4 ( "Field of the Invention : . . ."): 

— Cross-Reference to Related Applications : 

This application is a continuation of PCT/US0 1/04963 and claims priority to 
United States Patent Applications Serial Nos. 60/197,971 (filed April 17, 2000, now 
abandoned) and 60/183,588 (filed February 18, 2000, now abandoned), each of which 
applications is herein incorporated by reference in its entirety. 

3. Please replace the paragraph commencing on page 2, line 28 ("As photograph 
library sizes increase, the need and benefit. . .) with the following replacement 
paragraph: 

As photograph library sizes increase, the need and benefit of annotation and search 
capabilities grows. The need to rapidly locate photographs of, for example, Bill Clinton 
meeting with Boris Yeltsin at a European summit held in 1998, is strong enough to justify 
substantial efforts in many news agencies. More difficult searches such as images 
depicting "agriculture in developing nations" are harder to satisfy, but many web and 
database search tools support such searches (Lycos, Corbis, etc.). Query-By-Image- 
Content from IBM, is one of many projects that use automated techniques to analyze 
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image (http://wwwqbic.almadcn.ibm.com/) . Computer vision techniques can be helpful 
in finding photographs by color (sunsets are a typical example), identifying features 
(corporate logos or the Washington Monument), or textures (such as clouds or trees), but a 
blend of automated and manual techniques may be preferable. Approaches to the non- 
textual or "content-based" annotation of images are described in U.S. Patent Nos. 
5,91 1,139 (Issued June 8, 1999), 5,899,999 (Issued May 4, 1999), and 5,579,471 (Issued 
November 26, 1996) and PCT Applications WO09935596A1 (Published July 15, 1999), 
WO09931605A1 (Published June 24, 1999). The non-textual searching of image files is 
discussed by Rui, et al. (1999; "Image Retrieval: "Current Techniques Promising 
Directions and Open Issues[[,]]" http://citc s ccnnj>ncc>com/3 84 715.htinl ), and by Chang, 
S.-F. et al. (1997), "Visual Information Retrieval from Large Distributed On-line 
Repositories^,]]" http://citcsccr.nj.nec.com/ chang97visual.html) . Face recognition 
research offers hope for automated annotation, but commercial progress is slow (R. 
Chellappa, C.L. Wilson and S. Sirohey, "Human and Machine Recognition of Faces: A 
Survey" Proceedings of the IEEE, Vol. 83, pp. 705-740, May 1995; Allan Kuchinsky, 
Celine Pering, Michael L. Creech, Dennis Freeze, Bill Serra, Jacek Gwizdka, "FotoFile: A 
Consumer Multimedia Organization and Retrieval System", Proceedings of ACMCHI99 
Conference on Human Factors in Computing Systems, 496-503, 1999). 

4. Please replace the paragraph commencing on page 3, line 24 ("The annotation 
of photographs is a variation on previously explored. . .) with the following 
replacement paragraph: 

The annotation of photographs is a variation on previously explored problems such 
as annotation on maps (E. Imhof, "Positioning Names on Maps", The American 
Cartographer, 2, 128-144, 1975; J. Christensen, J. Marks, and S. Shieber, "An Empirical 
Study Of Algorithms For Point-Feature Label Placement", ACM Transactions on Graphics 
14, 3, 203-232, 1995; J. S. Doerschler and H. Freeman, "A Rule-Based System For Dense- 
Map Name Placement", Communications OfTheACM35, 1, 68-79, 1992) in which the 
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challenge is to place city, state, river, or lake labels close to the features. There is a long 
history of work on this problem, but new possibilities emerge because of the dynamics of 
the computer screen. Such efforts, however, are fundamentally different from those 
involved in the annotation of photographs, video, and other electronic images in which the 
image aspects (i.e., the objects, people scenes, depictions, etc. that are contained within an 
electronic image) are inherently unrelated to one another. Maps (and electronic timelines, 
s uch a s s hown http://chao s Lhypcrmart.net/ cgypt/tac.html ) are X,Y (or, in the case of 
timelines, X) representations of a surface or construct in which the relative placements of all 
desired annotations are determined solely by the scale of the image and the X,Y (or X) 
coordinate of its center, rather than by inspection and analysis of the image. The annotation 
of maps and timelines is possible using prior art methods because their image aspects are 
inherently related to one another. Annotation is usually seen as an authoring process 
conducted by specialists and users only chose whether to show or hide annotations. 
Variations on annotation also come from the placement of labels on markers in information 
visualization tasks such as in tree structures, such in the hyperbolic tree (John Lamping, 
Ramana Rao, and Peter Pirolli, "A Focus + Context Technique Based On Hyperbolic 
Geometry For Visualizing Large Hierarchies", Proceedings of ACM CHI95 Conference on 
Human Factors in Computing Systems, New York, 401-408, 1995) or in medical histories, 
such as LifeLines (Jia Li., Catherine Plaisant, Ben Shneiderman, "Data Object and Label 
Placement for Information Abundant Visualizations" Workshop on New Paradigms in 
Information Visualization and Manipulation (NPIV'98), ACM, New York, 41-48, 1998). ~ 



5. Please replace the paragraph commencing on page 4, line 23 ("Previous work 
on electronic image annotation focused on the writing. . .) with the following 
replacement paragraph: 

Previous work on electronic image annotation focused on the writing of computer 
programs to make label placements that reduced overlaps (Mark D. Pritt, "Method and 
Apparatus for The Placement of Annotations on A Display without Overlap", US Patent 
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5689717, 1997), but there are many situations in which it is helpful for users to place labels 
manually, much like a post-it® note, on documents, photographs, maps, diagrams, 
webpages, etc. Annotation of paper and electronic documents by hand is also a much- 
studied topic with continuing innovations (Bill N. Schilit, Gene Golovchinsky, and Morgan 
N. Price, "Beyond Paper: Supporting Active Reading with Free Form Digital Ink 
Annotations," Proceedings of ACM CHI 98 Conference on Human Factors in Computing 
Systems, v.l 249-256, 1998). While many systems allow notes to be placed on a document 
or object, the demands of annotating personal photograph libraries are worthy of special 
study (J Kelly Lee and Dana Whitney Wolcott, "Method of Customer Photoprint 
Annotation", US Patent 5757466, 1998). Personal photograph libraries are considered to 
represent a special case because users are concentrating on the photographs (and may have a 
low interested in the underlying technology), are concerned about the social aspects of 
sharing photographs, and are intermittent users. They seek enjoyment and have little 
patience for form filling or data entry. Personal photograph libraries may have from 
hundreds to tens of thousands of photographs, and organization is, to be generous, 
haphazard. Photographs are sometimes in neat albums, but more often put in a drawer or a 
shoebox. While recent photographs are often on top, shuffling through the photographs 
often leaves them disorganized. Some users will keep photographs in the envelopes they 
got from the photography store, and more organized types will label and order them. The 
annotating of such personal images is such a time-consuming, tedious and error-prone data 
entry task that it discourages many archivists and librarians from maximizing the value of 
their image collections. As noted by Rui, et al (1999 ; http://citc s ccr.nj.iicc. 
com/3 84 715. html ), the perception subjectivity and annotation impreciseness of text-based 
image retrieval systems may cause unrecoverable mismatches in the later retrieval process. 
The problem is particularly significant for most owners of personal photograph libraries. - 



